Recognizing activities with cluster-trees of tracklets

Although the structure of simple actions can be captured by rigid grids [3] or by sequences of short temporal parts [2], activities are composed of a variable number of sub-events connected by more complex spatio-temporal relations. In this paper, we learn how to automatically represent activities as a hierarchy of mid-level motion components in order to improve activity classification in real-world videos. This hierarchy is a video-specific, data-driven decomposition obtained by clustering tracklets, i.e. local point trajectories of a fixed small duration. Our first contribution is a hierarchical spectral clustering algorithm, based on top-down recursive bi-partitioning. We propose to robustly threshold tracklet projections on an approximate spectral embedding by minimizing a spatio-temporal connectivity cost. This allows for clusters of arbitrary shape and an efficient greedy splitting strategy that automatically determines the number of clusters. The resulting hierarchical decomposition provides structural information relating motion parts together. Our second contribution is the use of this entire tree structure, called cluster-tree (c.f. Figure 1), in order to build a hierarchical model of the motion content of a video. We introduce a corresponding tree representation of actions, called BOF-tree. The BOF-tree of a video has the same structure as its cluster-tree and each node is modeled by a bag-offeatures (BOF) over the MBH descriptors [6] of its constitutive tracklets. Efficiently using this structural information is challenging as cluster-trees have a variable number of nodes and a structure specific to each video. Furthermore, there is no natural left-to-right ordering of the two children of a parent node. Therefore, we introduce an efficient positive definite kernel — called the “All Tree Edge Pairs” (ATEP) kernel — that computes the structural and visual similarity of two hierarchical decompositions by relying on models of their parent-child relations as described in the following. Let Ti = (Vi,Ei), i ∈ {1,2}, be two BOF-trees, defined from their set of vertices (nodes) Vi and directed edges (parent-child relations) Ei. Each node v ∈ Vi is represented by a BOF — noted b[v] — over its constitutive tracklets. We model a directed edge e = (vp,vc) ∈ Ei by the concatenation — noted b[e] = (b[vp],b[vc]) — of the BOF of the child node vc with the BOF of its parent node vp. Let h be a kernel between BOF, ri ∈ Vi be the root of Ti, and wr ∈ (0,1) a cross-validated parameter encoding a prior on the importance of the root-to-root comparisons. Our ATEP kernel is defined as: